3. Multimodal Communication

1. Why is the topic of multimodal communication interesting? Face-to-face communication is the evolutionary primary and most common way of communicating for humans. It is not unlikely that vocal-verbal language with more or less intentional control, but still to some extent systematic, developed from being action-and gesture-based to being vocally based (Arbib 2005). In contrast to the hundreds of thousands of years of natural development of face-to-face communication, communication technologies have been artificially developed in order to supplement, enhance or replace certain types of face-to-face communication and today we have a situation of steadily increasing use of communication technology with possibilities for using multiple modalities. In this chapter, we will give an account of human-to-human multimodal communication, in order to have a background for a discussion of the use of multimodality in new communication technology (see also Lücking and Pfeifer as well as Gibbon both in this volume). We will primarily deal with face-to-face communication, rather than multimedia in relation to written communication. First, we turn to a brief overview of human-human communication in order to see what features can potentially be used in human-computer interaction in general and more specifically in dialog and tutoring systems. Then, we will discuss some problems in making computer based systems multimodal and thereby more human like. 2. Human-human communication 2.1. What is multimodal communication? The word " modal " has the fairly abstract meaning " pertaining to manner or mode " (e.g., Collins English Dictionary 2009), so it is not surprising that the term " multimodal communication " has been used in many different ways. For example, it has been used for the three phenomena that might more appropriately be called " multi-medial communication " , " multi-representational communication " and " multi-mass-medial communication " (cf. Allwood 2008). Here we will take " multimodal communication " to mean communication involving more than two of the sensory modalities (sight, hearing, touch, smell and taste). For practical purposes, we will also, besides " sensory or perception modalities " , talk about " production modalities " , by which we mean human bodily